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CKM mixing between tliird family quarks and a possible fourtli family is constrained 
by global fits to the precision electroweak data. The dominant constraint is from 
nondecoupling oblique corrections rather than the vertex correction to Z ^ 66 used in 
V/~) previous analyses. The possibility of large mixing suggested by some recent analyses of 
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FCNC processes is excluded, but 3-4 mixing of the same order as the Cabbibo mixing 
of the first two families is allowed. 



Introduction 

A fourth family of quarks and leptons is an obvious extension of the Standard Model 
(SM) that will be investigated at the LHC.[1] If a fourth family is discovered, it is hkely to 
have consequences at least as profound as those that have emerged from the discovery of 
the third family. The necessarily heavy neutrino mass, rriu^^ > would be surprising, 

but without a theory of neutrino masses we are not really in a position to judge; if a fourth 
family were discovered, it would instantaneously refocus the effort to understand neutrino 
masses (see for instance [2, 3]). A fourth family is consistent with precision electroweak 
(EW) data, [4, 5, 6] and can remove[5] the tension between the SM fit and the LEP II lower 
bound on tuh that arises if the 2>.2a discrepancy between hadronic and leptonic determina- 
tions of sin^6'^*'''^ turns out to be the result of underestimated systematic error. [7] The SM 
augmented with a fourth family is consistent with SU (5) gauge coiipling unification without 
supcrsymmctry. [8] Electroweak baryogenesis might be viable with four families, [9] although 
it is not in the SM with only three. Since the plausible parameter space includes the strong 
coupling region determined by perturbative unitarity, [10] mg^ ^ 550 GeV, a heavy fourth 
family could naturally play a role in the dynamical breaking of electroweak symmetry. [2, 11] 
Even if fourth family quarks are very heavy, e.g., itlqI^I TeV, and difficult or impossible 
to observe directly, they will give rise to a large nonresonant signal for production of longi- 
tudinally polarized Z boson pairs from gg ZZ^ that could be seen at the LHC with ba 
significance over backgrounds with only 0(10) fb~^ of integrated luminosity. [12] 

Motivated initially by an interesting study of the FCNC constraints on a unitary 4x4 
CKM matrix by Bobrowski et a/., [13] wc have studied the constraint on 3-4 family CKM 
mixing that can be obtained from precision electroweak data. They found, in addition to 
the expected small angle solutions, that surprisingly large mixing between the third and 
fourth family quarks is also allowed. They exhibit fits with | 1 / 1 V'jf ^^'"^ | as small as 0.73, 
corresponding to \Vtib\ as large as ~ 0.63, just at the edge of the 95% allowed region for \Vtb\ 
determined from single top production. [14] We find however that these fits are decisively 
excluded by the precision EW data and present the EW constraints on 3-4 family CKM 
mixing for a range of fourth family masses favored by the EW data. Although the large- 
mixing FCNC fits are excluded, the EW constraints do allow 3-4 CKM mixing of the same 
order as the Cabbibo mixing of the first two families. Our results are also inconsistent with 
large 3-4 mixing parameters obtained in another recent study [15] and constrain proposals to 
explain the CP anomalies suggested by B meson data. [16, 17] 

In the presence of 3-4 CKM mixing there are two nondecoupling radiative corrections 
to the precision EW observables with quadratic sensitivity to heavy fourth-family fermion 
masses: the p parameter correction, [18, 10] (AKA the obhque parameter q;T[19]) and the 
Zhb vertex correction. [20] In the SM both are proportional to Gprnf at one loop order. In the 
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four-family model they give rise to corrections proportional to \Vt'b\'^m^, and, in the case of 
the oblique corrections, there are also large corrections proportional to \ Vtb'\^ml, if ml, ^ m^. 

Previous consideration of the precision EW constraint on 3-4 CKM mixing focused on 
the effect of the vertex correction on Rb — V{Z — > bb)/r{Z — > hadrons),[21, 22] which was 
used as a constraint in subsequent FCNC studies (e.g., [16, 17]). However the obhque cor- 
rections are of precisely the same order and actually provide the most important constraint. 
To obtain a valid bound it is essential to reevaluate the global EW fit in the new physics 
model, since the not infrequently followed practice of using just the magnitude of shifts from 
the values in the SM fit does not take into account the possibihty that a global fit incor- 
porating the new physics may have its minimum at significantly different values of the 
SM parameters, e.g., rrit and, especially, mn-^ This in fact occurs in the results presented 
below. In the fits that establish the 95% CL limits (e.g., tables 2 and 5) Rb is only la from 
its experimental value and docs not contribute to the constraint.'^ In addition, global fits are 
less susceptible to statistical fluctuation or systematic uncertainty than a constraint based 
on a single observable. For instance, the 2o" upper limit on 3-4 mixing obtained from the 
(negative) shift in Rb from its SM value would have been significantly weaker if the SM value 
for Rb were 0.7a above the experimental value rather than 0.7a below. 

In the following sections wc briefly review the nondccoupling corrections and present 
the bounds on the mixing angle that follow from the global flts. Because the heaviest quark 
masses considered are at the threshold of the strong coupling region, we use the two loop 
correction to the p parameter as a guide to the applicability of perturbation theory and the 
accuracy of the results. 
Nondecoupling corrections 

For vanishing CKM angles, the one loop correction to the p parameter from a heavy 
fermion doublet (/i,/2) is [18, 10] 

Sp^Nc^ ^ (1) 

where Nq = 1, 3 for leptons and quarks respectively, xw — sin^Ow, and F12 is 

X1+X2 X1X2 , Xi 

F12 = — In— 2) 

2 Xi- X2 X2 

with Xi = m1/m\. To study the EW constraints on the large-mixing parameter sets of 

^Alwall et al. consider the oblique constraints and remark that they are sensitive to uih. They do not 
perform a global fit, which would incorporate the niH dependence, and instead rely on for their strongest 
precision EW constraint. 

^The shift in Ri, is dominated by the nondecoupling vertex correction. The oblique corrections to T{Z — > 
hadrons) and T{Z bb) are significantly larger but cancel in the ratio. T{Z hadrons) and T[Z bb) are 
not included in the fits but can be derived from combinations of observables that are. 
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[13] it suffices to assume a block-diagonal form for the 3-4 CKM submatrix,^ which is then 
characterized by a single angle, ^34, with \Vtb\ — \Vt'b'\ — C34 and |Vt/b| = |T46'| = •534, where 
C34 = cos^34 and S34 = sin^34. The oblique correction T from the fourth family is then 

T4 — 7, y-^ r (3 \Ft'b' + S34(-ft'6 + Ftb' — Ftb — Ffb') + -^^1/4) • (3) 

The term —s'^^Ftb is the decrease from the three-family SM tb contribution to the W boson 
vacuum polarization. Ffb is the largest term and puts the strongest constraint on 6*34. 
Equation (3) is easily obtained following the derivation for Vtb = 1 in the second paper 
cited in [10]; for Vtb 7^ I the GIM mechanism and the custodial SU{2) together ensure that 
the divergences cancel between the W and Z vacuum polarization terms, leaving the finite 
correction in (3). 

We also need the oblique correction S for the fourth family quark doublet, which to a 
very good approximation [4] is given by 

'l - h/-] . (4) 



Gtt V ?) X2, 

For the lepton masses considered below, the leptonic contribution to S is negligible. 

Finally there is the nondecoupling correction to the Zbb interaction of the left handed 
h quark, which arises in the SM from one loop vertex corrections containing the t quark and 
the W boson. In our notation the interaction Lagrangian is 

C^^—gbLbL^hL (5) 
cosOw 

where in the SM gbL = ~i + \xw- In the SM the one loop, nondecoupling vertex correction 
from the t-\-W loop graphs is [20] 

= ^ V 4 (6) 

The one loop correction from 3-4 CKM mixing is then 

.V- 3-4 _ 2 ^ /ml _ m[\ 

d 9bL - ^34 ^g^^^(^ _ (^^2 

where again the last term accounts for the decrease of the SM top quark correction. 
Bounds on ^34 

Like Bobrowski et a/. [13] we focus on parameters for the fourth family shown by Kribs 
et a/. [6] to be favored by the precision EW data. In addition to the three parameter sets 



■^We have verified explictly for the large-mixing parameter sets of [13] that this approximation is valid to 
better than 1% for the diagonal matrix elements and to better than 3% for the off-diagonal ones. 
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Table 1: The three large-mixing parameter sets of [13] with the corresponding values of T4 
from equation (3) and the for 12 degrees of freedom from the global fits. The pulls of 
four sensitive observables are compared with the pull of it!^. 

identified by Bobrowski et ai, shown in table 1, we survey other t' masses between 300 and 
600 GcV. Following [5] and [6] we confirm for S34 — that is minimized for \mt' — mbi\ ~ 
45 — 75 GeV and set the b' mass to 

rub' — mt' — 55 GeV. (8) 

As discussed below, the limits on S34 do not depend sensitively on this choice. The lepton 
masses, which have relatively little effect on the limit on S34, are chosen as rrii,^ = 100 GeV 
and rrii^ = 145 GeV.^ 

Table 1 shows that the three large-mixing parameter sets of [13] are excluded "with 
extreme prejudice" by the EW data. The large contributions to T4 are responsible for the 
huge values. In these fits the dominant contributors to x^ ^'I'g the W boson mass, the Z 
boson width, and one or both of A^p^and Alr. Since the FCNC constraints for these three 
parameter sets have been carefully considered in [13], we include them in the 95% CL limit 
fits presented below. 

Following the procedure of the EWWG,[23] the global fits are obtained by minimizing 
while varying four SM parameters: m^, Aas, as-, and iriHi with itih allowed to vary freely 
between 10 GeV and 1 TeV. Like the EWWG we leave 0:5 unconstrained and determine it 
from the fits; for all the fits at or within the 95% CL limit for IS34I, the values of 0:5 are in 
reasonable agreement with other determinations. The global constraints on ^^34 are fairly 
valued because they allow for the possibility of x^ minima at points in the parameter space 
that are quite different than the location of the SM minimum, and they are efficient because 
they aggregate the effect of the new physics on all of the relevant observables. 

Radiative corrections for the global fits are computed with ZFITTER[24], including the 
two loop corrections to sin^^^^*^^ [25] and mvi/-[26] Our fits include the largest experimental 
correlations, taken from the EWWG. When we use the same measurement set, our SM fit 

^We assume a Dirac mass for v^. A dynamically generated Majorana mass for the fourth neutrino which 
made a negative contribution to T could weaken the constraints for given mc . [3] 
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agrees closely with the EWWG. Unlike typical fits with the obhque parameters S and T, 
which are performed with respect to fixed reference values of the SM parameters, we use the 
complete set of radiative corrections from ZFITTER to compute the dependence of oii 
the SM parameters ttih, irit, Acts, and as, for each {S4, T4) pair, which represent only the 
fourth family corrections. This procedure is then more accurate, however the resulting S, T 
values are not directly comparable to S and T from typical fits with fixed "reference" values 
of the SM parameters. 

In this work we focus on the EWWG set of observables.^ We also briefiy describe the 
results for the data set without the three hadronic front-back asymmetry measurements. [7] 
The fits are performed for four-family models with m^/ between 300 GeV and 1 TeV, including 
the three parameter sets of [13], with h' and lepton masses as described above. In all cases 
the minimum occurs at ^34 = 0. The 95% CL upper limit is obtained by increasing 
^34 until increases by 3.84 units, corresponding to CL(Ax^, 1) = 0.95, like the procedure 
commonly used to obtain the 95% upper limit on uih in the SM fits.^ The fits at mt' = 1 TeV 
are intended primarily to probe the range of applicability of perturbation theory. 

The results for m^/ = 500 GeV arc illustrated in figure 1 and table 2. Tabic 2 displays 
the SM fit, the four-family fit for m^/ = 500 GeV and ^^34 = and also for ^^34 at its (one 
loop) 95% upper limit, IS34I = 0.15. cis a function of IS34I is shown in figure 1. The x^ of 
the four family model with ^^34 = is little changed from the SM but the central value of mn 
is increased, from 85 to 139 GeV, and the 3.2(T discrepancy between A^r and is more 
equally shared between the two, in contrast to the SM fit in which is the outlier.'' In 
the fit at the edge of the 95% confidence level, mn increases to the strong coupling regime 
near 1 TeV because of the increased value of T4, and Alr has become the outlier, while the 
pull of Rb is only 1.2. The central value of itlh as a function of |sin^34| is shown in figure 2. 

In table 3 wc summarize the limits on ^^34 for seven values of mti between 300 to 650 GeV, 
including the three parameter sets from reference [13], and in addition at m^/ = 1 TeV. In 
all cases the fits at ^^34 = are nearly identical to the fit shown in table 2 for m^/ = 500 GeV. 
As we would expect from equations (3) and (7), the limit on |sin6'34| becomes proportional 
to l/mt' for iTLti ^ nit. For these fits at the 95% confidence limit, the Higgs boson mass is at 
mu = 790±30 GeV and T4 = 0.47±0.01. In all cases we have \Vtb\ ~ \Vt'b'\ ^ |cos^34| > 0.94 
and for nit' > 500 GeV we have |cos^34| > 0.99. 

^Unlike the EWWG wc do not include the W boson width, which with a 2% error is not a precision 
measurement in the sense of the other measurements that typically have part per mil precision. In any case, 
Tw has a negligible effect on the fits. 

^For the SM fits the limit is at A^^ = 2.71 corresponding to the 90% symmetric confidence interval and 
the 95% upper hmit. Because the bound on |6'34| is one-sided, the 95% limit is at Ax^ = 3.84. 

^We assign 12 degrees of freedom to each four-family fit, assuming discovery at the LHC as a prior. This 
convention has no effect on the 95% CL limit. 
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rrit- = 500 GeV 
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sin 

Figure 1: distribution as a function of |sin^34| for the global fit to the four family model 
with rrit' = 500 GeV. The horizontal line indicates the 95% confidence interval. 
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0.05 





Table 2: Global fits: the SM, the 4 family SM with mf = 500 GeV and S34 = and again 
with S34 at the 95% confidence level. 
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Figure 2: Higgs boson mass as a function of |sin^34| for the global fit to the four family model 
with rrit' — 500 GeV. The horizontal hne indicates the 95% confidence interval for |sin^34|. 
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Table 3: 95% CL upper hmits on [534 1 at one and two loops from global fits to the EWWG 
data set. T4 and niH from the 95% CL fits are also shown. 
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To gauge the range of applicability of perturbation theory for large quark masses, the 
limits in table 3 have been obtained at both one and two loop order in the leading, non- 
decouphng electroweak corrections to the p parameter.^ The leading two loop corrections 
for large quark mass are known, [27, 28, 29] but in no case with both the Higgs boson mass 
dependence, which is large, and the dependence on the masses of both quarks in the doublet. 
The two loop corrections computed by Barbieri et a/., [28] are best suited for our purpose: 
they include the full mn dependence but for only one heavy quark in the doublet, i.e., for 
tuqi ^ TOQ2 — 0. This captures the contribution that is most important for the bound on 
the mixing angle, because the sin6'34 dependent term in T4, equation (3), is dominated by 
Fj'ft, for which it is always an excellent approximation. 

We do not need to consider the two loop correction to F^/^/, because even the one loop 
term has a negligible effect on the sin^34 dependence of T4. The Ftb' term is however somewhat 
problematic. The two loop correction, F^y , can safely be neglected for the smallest masses 
we consider, say mt/^400GeV, for which the one loop term F^y is not very important, 
but for 500 GeV ^rrif TeV, F^l) is not negligible and approximating 1111,1 = in the two 

loop correction F^^) may be a poor approximation. To give a conservative indication of the 

{'2) 

possible error, the effect on the bounds of a ±100% variation in the value of F^y is shown 

(2) 

as iAJj, in table 3. The conclusion is that the bounds on sin^^34 are probably good to a 
few percent at m^/ = 300 GeV, while at m^/ = 650 GcV they arc probably valid to from 10 

to 20%. At niti = 1 TcV the order one variation from A\J significantly overestimates the 

(2) 

uncertainty from F^y , but the order 50% shift from the first to second order limit on |s34|, 
from 0.07 to 0.11, is a signal that perturbation theory has become unreliable. 

The upper limits do not depend sensitively on the choice of mf — niy in equation (8). 
For larger mass differences, e.g., mt' — my ^ 100 GeV, there are no aceptable fits: at ^^34 = 
the fits arc poor, with CL(x^) < 0.03, and quickly become much poorer as |^^34| increases. 
For smaller mass splitting, e.g., at the extreme, m^/ = m;,/, the CL's are acceptable for 
^34 = 0, with CL(x^) = 0.18, but the predictions for the Higgs mass are unacceptable, with 
niH — 35 GeV and CL(m// > 114) = 0.0016. In this case marginally acceptable fits can 
be found by increasing IS34I, which raises the Higgs mass toward the allowed region while 
maintaining acceptable CL's. For instance, with m^/ = my — 500 GeV and S34 = 0.07, 
the fit is just consistent at 5% CL with the 114 GeV lower limit on mn, with mn — 58 GeV, 
Ch{mH > 114) = 0.05 and GL(x^) ^ 0.18. Taking Ax^ = 3.84 from either this fit or from 
the fit at ^34 = 0, we find the one loop 95% upper hmit at IS34I < 0.155, little changed 
from the limit that we obtained using m^/ — my — 55 GeV. There is a simple reason for the 
insensitivity of the limit to the mass splitting: as |s34| inceases, the terms in equation (3) 

^We have not included the two loop correction to the Zhb vertex, since the nondecoupling vertex correction 
does not have an important effect on the global fits. 
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nit' = 500 GeV 
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Figure 3: distribution as a function of |sin^34| of the global fit to the four-family model 
with rrit' — 500 GeV, for the data set without the hadronic asymmetry measuements. The 
horizontal line indicates the 95% confidence interval. 
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Table 4: 95% CL upper limits on |s34| at one and two loops from global fits to the EWWG 
data set without the hadronic asymmetry measurements. and ttih from the 95% CL fits 
are also shown. 

that control the limit are Ffb and F^^/ while Fj/^', which is sensitive to mf — my, plays a 
relatively minor role. 

It is interesting to consider the data set with hadronic asymmetries excluded, moti- 
vated by the possibility that underestimated sytematic error might contribute to the 3.2cr 
discrepancy in the SM determination of sin^^^^'^^ from the hadronic and leptonic asymme- 
try measurements. [7] The hadronic asymmetry measurements are more challenging, both 
experimentally and, especially, theoretically, because of the difficulty of extracting quark 
asymmetries from the actual measurements of hadronic final states. The QCD corrections 
are large, three times the experimental error, and the Monte Carlo calculations needed to 
merge the large QCD corrections with the experimental cuts give rise to a systematic error 
that is difficult to quantify. Without the three hadronic measurements, A^^, A^pb-i ^^'^ Qfb, 
the confidence level of the SM fit improves dramatically but the fit predicts a very fight Higgs 
boson, niH — 50 GeV, with 95% upper fimit at 105 GeV, in confiict at 97% CL with the 
114 GeV LEP II lower limit. Results are shown in figure 3 and tables 4 and 5. As found by 
Novikov et al.[5], the four-family model removes the confiict with the LEP II lower limit for 
this data set. The fimits in table 4 are ~ 40% stronger than for the full data set in table 3. 
At the 95% fimit for 634 the Higgs boson masses range from 275 to 300 GeV with T4 — 0.35 
in all cases. Because T4 and mn are smaller for this data set, both the one loop and two 
loop corrections are smaller, and perturbation theory appears to be under better control. 

Discussion 

While this work was initially motivated by the paper of Bobrowski et al, several other 
studies have considered the possible role of a fourth generation on FCNC phenomena and 
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Table 5: Global fits for the data set without the hadronic asymmetry measurements: the 
SM, the 4 family SM with mf — 500 GeV and S34 = 0, and again with S34 at the 95% 
confidence level. 
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the CKM matrix. The fits of Yanir[21] for m^/ = 500 GeV require IS34I ^0.14 and therefore 
fall within the 95% CL hmit of the fit to the complete data set, table 3, and just beyond the 
hmit from the fit with hadronic asymmetries excluded, table 4. The 95% CL limit quoted 
by Alwall et al, \cu\ > 0.93, [22] is weaker than the global fit hmits in tables 3 and 4, 
especially for the larger values of Wf. Herrera et al.[15\ have studied the FCNC constraints 
in texture models of the 4x4 quark mass matrices. They remark on an isolated solution 
with \Vtb\ ~ 0.88, implying IS34I ~ 0.47, which is decisively excluded by the EW fits, like 
the parameter sets in table 1. Most of their fits lie within 0.90 < \Vtb\ < 0.94, which is 
excluded at 95% CL in all cases considered here except the fit to the complete data set with 
mf = 300 GeV shown in table 3. 

Hou et a/. [16] and Soni et a/. [17] have identified regions in the CKM4 parameter space 
that could explain possible anomalies in B meson CP measurements, requiring large mixing 
between not only the third and fourth families but also between the second and the fourth. 
For instance, Hou et al. consider the four family model with rrit' = 300 GeV, S34 = 0.22, 
and with 2-4 mixing given by IV^'^I = 0.114 and \Vcb'\ = 0.116. There are then nonnegligible 
contributions to T4 from the 2-4 family mixing and additional terms must be added to 
equation (3), which becomes 



+ 



87rx^(l — xw) 



Using the above mixing angles and m*/ = 300 GeV the fit to the full data set yields 
Ax^ = 2.44, which falls within the 95% CL hmit at Ax^ = 3.84. For the fit with hadronic 
asymmetries excluded we find Ax^ = 4.64, just beyond the 95% CL limit. 

Soni et al.[n\ quote a range of values for the product A|, = for values of rrit' 

between 400 and 700 GeV. The limit on A|, then depends on the hierarchy between the CKM 
matrix elements, 

(10) 

Neglecting a possible contribution which might be expected from Feb' but is not specified 
in [17], the limits quoted in tables 3 and 4 for now apply instead to the combination 
(1 -|- r^)s§4. The corresponding upper limit on X^, = rs\^ is then a function of r, 

K'<Y^,x,, (11) 

where X95 is the square of the 95% CL upper limits on S34 given in the tables. The limit 
is maximal for r = 1, corresponding to Aj/ < X95/2; in this case the full range of preferred 
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values in [17] is allowed for the fit to the full data set and only shghtly restricted in the fit 
with hadronic asymmetries excluded. If we assume a hierarchy of order the Cabibbo angle, 
r ~ 0.2, then the bound is tighter, Xp ^Xg^/b; in this case a portion of the preferred range 
in [17] is excluded at 95% CL for both data sets, but a significant portion continues to be 
allowed. 

If a fourth family were discovered at the LHC, the subsequent study of its properties 
would be a major undertaking, with many profound implications. The elucidation of the 
four family CKM matrix would be important to understand the on-shell measurements at 
the LHC as well as the virtual implications for flavor physics and CP violation. Electroweak 
precision data, perhaps eventually augmented by a second generation Z boson factory, should 
continue to play an important role, by constraining and vetting the emerging picture and 
even by indicating a mismatch with direct high energy measurements that could be a signal 
for still unobserved new physics. 
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